Cooperative lock override procedure

ABSTRACT

Queued lock services for managing a shared resource in a data processing system include a cooperative lock override procedure. On detecting a protocol failure by another processor, the detecting processor confirms that the failing processor is the lockholder and passes the lock to the next requestor in the queue.

FIELD OF THE INVENTION

[0001] This invention relates generally to a method and apparatus forimproving performance in systems where multiple processors contend forcontrol of a shared resource through a lock associated with the sharedresource, and more particularly to a method and apparatus for improvingperformance in intelligent data storage systems.

BACKGROUND OF THE INVENTION

[0002] When a computer system resource is shared by multiple processesrunning on multiple processors, or even on one processor, often theremust be some way of insuring that no more than one such process mayaccess that resource at any one time. In designing complex data storagesystems including multiple processors, synchronizing access to sharedresources has been recognized as an issue which must be addressed inorder to maintain the consistency and validity of the data. However, thesharing issue may arise in connection with almost any resource thatmight be used by multiple requesters.

[0003] Many high-performance storage systems are intelligent datastorage systems which may be accessible by multiple host computers.These may include, in addition to one or more storage device arrays, anumber of intelligent controllers for controlling the various aspects ofthe data transfers associated with the storage system. In such systems,host controllers may provide the interface between the host computersand the storage system, and device controllers may be used to manage thetransfer of data to and from an associated array of storage devices(e.g. disk drives). Often, the arrays may be accessed by multiple hostsand controllers. In addition, advanced storage systems, such as theSYMMETRIX® storage systems manufactured by EMC Corporation, generallyinclude a global memory which typically shared by the controllers in thesystem. The memory may be used as a staging area (or cache) for the datatransfers between the storage devices and the host computers and mayprovide a communications path which buffers data transfer between thevarious controllers. Various communication channels, such as busses,backplanes or networks, link the controllers to one another and theglobal memory, the host controllers to the host computers, and the diskcontrollers to the storage devices. Such systems are described, forexample, in Yanai et al, U.S. Pat. No. 5,206,939 issued Apr. 27, 1993,(hereinafter “the '939 patent”), Yanai et al, U.S. Pat. No. 5,381,539issued Jan. 10, 1995, (hereinafter “the '539 patent”), Vishlitzky et al,U.S. Pat. No. 5,592,492 issued Jan. 7, 1997, (hereinafter “the '492patent”), Yanai et al, U.S. Pat. No. 5,664,144 issued Sep. 2, 1997(hereinafter “the '144 patent), and Vishlitzky et al, U.S. Pat. No.5,787,473 issued Jul. 28, 1998, (hereinafter “the '473 patent”), all ofwhich are herein incorporated in their entirety by reference. Thesystems described therein allow the controllers to act independently toperform different processing tasks and provide for distributedmanagement of the global memory resources by the controllers. This highdegree of parallelism permits improved efficiency in processing I/Otasks. Since each of the controllers may act independently, there may becontention for certain of the shared memory resources within the system.In these systems, the consistency of the data contained in some portionsof global memory may be maintained by requiring each controller to lockthose data structures which require consistency while it is performingany operations on them which are supposed to be atomic.

[0004] Since locking inherently reduces the parallelism of the systemand puts a high load on system resources, locking procedures must bedesigned with care to preserve system efficiency. Adding features to thelock, such as queuing, lock override procedures, or multimodality canhelp to avoid some pitfalls of common lock protocols, such as processorstarvation, deadlocks, livelocks and convoys. However, it is also knownthat, while many of these lock features have individual advantages,multifeatured lock management procedures are difficult to design andimplement without unduly burdening system resources or inadvertentlyintroducing pitfalls such as additional deadlock or starvationsituations. For example, multimodal locks, which permit the requestor toidentify the kind of resource access desired by the requestor and thedegree of resource sharing which its transaction can tolerate, can beuseful in improving system performance and avoiding deadlocks, butproviding a lock override which is suitable for a multimodal lock isquite difficult. If, for example, one lock mode is set to allowunusually long transactions, a timeout set to accommodate normaltransactions will cut the long ones off in midstream while a timeout setto accommodate the long transactions will allow failures occurringduring normal transactions to go undetected for excessively longperiods. Moreover, timeouts are competitive procedures which, in certaincircumstances, undesirably offset the cooperative advantages of a queuedlock. Because of the complexities introduced by multifeatured locks, itis desirable to validate features and modes which create particularlysignificant drains on system resources, such as long timeout modes, butintroducing additional validation features can itself load systemresources to the point where the system efficiency suffers.

[0005] Providing suitable procedures becomes especially difficult incomplex multiprocessor systems which may contain a number of queuedlocks associated with different shared resources and where a requestormay have to progress through a number of lock request queues in turn inorder to complete a process. In these systems, it is desirable thatwhatever procedure is implemented be fair, ensure that each requestoreventually obtains access to the lock whether or not all otherrequestors in the system are operating properly, and minimize theaverage waiting time for each requestor in the queue to improve systemefficiency. Queued locks which implement a first-in-first-out (FIFO)protocol meet the fairness criteria because denied requests are queuedin the order they are received. One such lock services procedure, oftenknown as the “bakery” or “deli” algorithm, is described, for example, in“Resource Allocation with Immunity to Limited Process Failure”. MichaelJ. Fischer, Nancy A. Lynch, James E. Burns, and Alan Borodin, 20^(th)Annual Symposium on Foundations of Computer Science, San Juan, PuertoRico, October 1979, p 234-254; and “Distributed FIFO Allocation ofIdentical Resources Using Small Shared Space”, ACM Transactions onProgramming Languages and Systems, January 1989, 11(1): 90-114. When allrequestors in the system are operating properly, the basic “deli”algorithm also meets the other criteria, but a protocol violation suchas the failure of any processor in the lock request queue can lead tototal system deadlock. However, in all complex multiprocessor systems,occasional protocol violations are inevitable, and the “deli” algorithmmakes no provision either for detecting these through validationprocedures or otherwise, or for handling them when they occur. Moreover,the basic “deli” lock is a unimodal lock.

[0006] A lock is needed which supports multiple locking modes and makesprovision both for validation features to detect protocol violations andlock override procedures to manage the violations without undulyreducing system efficiency, and which also meets desirable designcriteria for fairness, wait time minimization and guaranteed access.

SUMMARY OF THE INVENTION

[0007] In accordance with the present invention, a lock mechanism formanaging shared resources in a data processing system is provided.

[0008] In accordance with the present invention, a method for providingqueued locking and unlocking services for a shared resource is provided.The services include a cooperative lock override procedure. In oneaspect, the locking services are multimodal and the cooperative lockoverride procedure is selectively associated with a lock mode.

[0009] In another aspect of the invention, a method for providingself-validating, queued lock services for managing a shared resource ina data processing system services includes providing a cooperative lockoverride procedure. The data processing system includes a plurality ofprocessors as lock requestors. Each processor supports atomic operationsand is coupled to the shared resource through one or more first commoncommunication channels. The method includes providing for each sharedresource an associated main lock data structure stored in a sharedmemory accessible by the plurality of processors. The main lock datastructure includes in a single atomic structure, the resources needed tolock the shared resource by a successful lock requester, to establish aqueue of unsuccessful lock requestors, and to validate the existence ofthe lock. Resources are also provided to validate the identity of thesuccessful lock requestor in connection with certain transactions. Themethod also includes providing for each shared resource, an associatedauxiliary lock data structure stored in a shared memory accessible bythe plurality of processors. The auxiliary lock data structure may be asingle entry, the entry being a single atomic structure, or it may be anarray which includes an entry for each processor, each entry being asingle atomic structure. Each entry includes the resources needed toidentify the successful lock requestor's place in a queue of requestersand to identify the successful lock requestor. Each entry may alsoinclude the resources needed to save a timestamp as a reference value.The method also includes providing for each processor a monitoringprocedure for detecting a predetermined indication of protocol failureby an one of the plurality of processors and identifying the failingprocessor. The method also includes providing for each processor a lockservices procedure including a queuing procedure for unsuccessful lockrequesters, locking and unlocking procedures for locking and unlockingthe shared resource by a successful lock requestor, and a cooperativelock override procedure responsive to the detection of the predeterminedindication of protocol failure. The method also includes detecting, byone of the processors, one of these predetermined indications ofprotocol failure and identifying the failing processor. The method alsoincludes, in a single atomic operation, examining the contents of theauxiliary lock data structure by the detecting processor to determinewhether the identified failing processor is the successful lockrequestor, and either, if the identified failing processor is thesuccessful lock requestor, in a single atomic operation by the detectingprocessor, examining the contents of the main lock data structure andwriting data to the main lock data structure to reserve the lock to thenext requestor in the queue of unsuccessful lock requestors and torevalidate the lock, or, if the identified failing processor is not thesuccessful lock requester, exiting the cooperative lock overrideprocedure.

[0010] Prior to the step of examining the contents of the main lock datastructure by the detecting processor, one of the requesting processorsmay, in a single atomic operation, examine the contents of the main lockdata structure to determine if another requesting processor haspreviously locked the shared resource and if the lock contents arevalid, and if it determines that the contents are invalid or no otherrequesting processor has previously locked the shared resource, it maywrite data to the main lock data Structure to reserve and validate thelock.

[0011] In one aspect of the invention, the lock services procedurefurther includes at least two lock mode procedures and a lock modeselection procedure for selecting one from the lock mode procedures by asuccessful lock requestor. The locking and unlocking procedures includeone or more procedures for locking and unlocking the shared resource inthe selected lock mode by a successful lock requester and thecooperative lock override procedure is selectively associated with alock mode. The atomic main lock data structure further includes theresources needed to identify one of the lock modes and the auxiliarylock data structure further includes the resources needed to identifyone from the lock modes. In examining the contents of the main lock datastructure, the detecting processor may, in the same atomic operation,verify that the identified lock mode is a lock mode associated with thecooperative lock override procedure and in writing data to the main lockdata structure to reserve the lock to the next requestor in the queue ofunsuccessful lock requestors the detecting processor may, in the sameatomic operation, invalidate the identified lock mode.

[0012] In another aspect, the invention provides an intelligent datastorage system. The intelligent storage system typically includesmultiple processors as requestors, and these are coupled to a sharedresource through one or more first common communication channels. Thesystem also includes a shared memory accessible over one or more secondcommon communications channels to all of the processors. Each processorsupports atomic operations. Each processor implements a monitoringprocedure for detecting a predetermined indication of protocol failureby a one of the plurality of processors and identifying the failingprocessor. A lock services procedure is also implemented in each of theprocessors. The lock services procedure includes a queuing procedure forunsuccessful lock requestors, and locking and unlocking procedures forlocking and unlocking the shared resource by a successful lockrequester, and a cooperative lock override procedure responsive to thedetection of the predetermined indication of protocol failure. M atomicmain lock data structure, responsive to the lock services procedures, isimplemented in the shared memory and associated with the sharedresource. The main lock data structure includes the resources needed tolock a shared resource by a successful lock requestor, to establish aplace in a queue of unsuccessful lock requesters, and to validate theexistence of the lock. An atomic auxiliary lock data structure,responsive to the lock services procedures, is also implemented in theshared memory and associated with the shared resource. The auxiliarylock data structure includes the resources needed to identify thesuccessful lock requestor's place in a queue of requesters and toidentify the successful lock requesters. Each processor is operable inaccordance with its monitoring procedure to detect a predeterminedindication of protocol failure and identify the failing processor. Eachprocessor is also operable in accordance with its lock servicesprocedure, first to initiate its cooperative lock override procedureresponsive to its detection of the predetermined indication of protocolfailure, and then in a single atomic operation, to examine the contentsof the auxiliary lock data structure to determine if the identifiedfailing processor is the successful lock requestor, and either, if theidentified failing processor is the successful lock requester, in asingle atomic operation, to examine the contents of the main lock datastructure and write data to the main lock data structure to reserve thelock to the next requestor in the queue of unsuccessful lock requestorsand to revalidate the lock, or, if the identified failing processor isnot the successful lock requester, to exit the cooperative lock overrideprocedure.

[0013] Each of the requesting processors is also operable in accordancewith its lock services procedure, in a single atomic operation, toexamine the contents of the main lock data structure to determine ifanother requesting processor has previously locked the shared resourceand if the lock contents are valid, and if it determines that thecontents are invalid or no other requesting processor has previouslylocked the shared resource, it may write data to the main lock datastructure to reserve and validate the lock.

[0014] In one aspect of the invention, the lock services procedurefurther includes at least two lock mode procedures and a lock modeselection procedure for selecting one from the lock mode procedures by asuccessful lock requester. The locking and unlocking procedures includeone or more procedures for locking and unlocking the shared resource inthe selected lock mode by a successful lock requestor and thecooperative lock override procedure is selectively associated with alock mode. The atomic main lock data structure further includes theresources needed to identify one of the lock modes. In examining thecontents of the main lock data structure, the detecting processor may,in the same atomic operation, verify that the identified lock mode isthe lock mode associated with the cooperative lock override procedureand in of writing data to the main lock data structure to reserve thelock to the next requestor in the queue of unsuccessful lock requestorsthe detecting processor may, in the same atomic operation, invalidatethe identified lock mode.

[0015] In yet another aspect of the invention, multiple processesrunning on a single processor may in some aspects act as requesters, anda lock allocation process or procedure may be invoked by each of theseprocesses, but the operation of the invention is otherwise as describedabove.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and further advantages of the present invention may bebetter understood by referring to the following description taken intoconjunction with the accompanying drawings in which:

[0017]FIG. 1 is a block diagram of a computer system including a sharedresource and incorporating the present invention.

[0018]FIG. 2 is a more detailed block diagram of the computer system ofFIG. 1 including an intelligent mass storage system.

[0019]FIG. 3 is a schematic diagram of the main lock data structure usedto implement the invention in the system described herein.

[0020]FIG. 4 is a schematic diagram of the auxiliary lock data structureused to implement the invention in some aspects of the system describedherein.

[0021]FIG. 5 is a flowchart illustrating steps used to enter the lockrequest queue, and to poll for and obtain the lock during normaloperation of the system described herein.

[0022]FIG. 6 is a flowchart illustrating steps used to perform a timeoutlock override procedure associated with a selected one of the lock modesimplemented in the system described herein.

[0023]FIG. 7 is a flowchart illustrating steps used to take the lock, toselect one of the lock modes, to perform a supplemental validationassociated with the selected lock mode, and to initialize a cooperativelock override procedure associated with the selected lock modeimplemented in the system described herein.

[0024]FIG. 8 is a flowchart illustrating steps used to perform thecooperative lock override procedure corresponding to a selected one ofthe lock modes in the system described herein.

[0025]FIG. 9 is a flowchart illustrating steps used to unlock the lockduring normal operation of the system described herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0026] Referring now to FIG. 1, computer system 10 is shown to include,among other things, a plurality of processors 1 a-1 n, running processesA-N, coupled to a shared resource 4 via one or more first commoncommunication channels 3 a-n and to a shared memory 2 via one or moresecond common communication channels 7 a-n. For purposes ofillustration, only one first common communication channel 3 and onesecond common communication channel 7 are shown in FIG. 1. Any or all ofprocessors 1 a-1 n may request access to shared resource 4 in order toexecute their processes A-N. The processors are actual or virtualdigital processing units which include one or more CPU's and additionallocal memory 5 a-n. For example, processor 1 a may be an intelligentdevice controller, an open-systems computer, a personal computer, aserver, an intelligent host controller or a virtual system residing on amainframe computer. Since each of the computer systems just mentionedtypically communicates using a specific communication protocol, each ofthe first and second common communication channels will correspondinglybe those channels specific to the computer system to which they arecoupled. That is for example, assuming processor 1 b is an open-systemstype server (e.g. running the UNIX Operating System), channel 3 or 7would typically be a SCSI type communications bus or a fibber-channelcommunications path. All communications over channel 3 or 7 wouldtherefore adhere to the respective SCSI or fibre-channel communicationsprotocols. Processes A-N may be, for example, procedures run by theprocessors, operating system processes or higher level applications. Theprocessors may run other processes not involving shared resource 4. Theinvention may also be applicable to multiple processes contending for ashared resource but running on a single processor, although this aspectis not illustrated in the drawings.

[0027] To synchronize accesses to the shared resource 4 and provide dataconsistency, system 10 also provides a queued lock associated withshared resource 4. The queued lock is implemented by a main lock datastructure, 30 and, in some aspects, an auxiliary lock data structure,40, both her described below, in shared memory 2 and a lock servicesprocedure 6 a-6 n running on each of processors 1 a-1 n, respectively.The lock data structures, 30 and 40, must be implemented in a section ofmemory that is accessible by all of the processors which might needaccess to the shared resource, although they need not be on the samemedia as the shared resource. The procedures which allocate the lock maybe centralized or distributed. In the intelligent data processingsystems described above, the lock services procedures are typicallydistributed among the various intelligent controllers.

[0028] The main lock data structure, 30, is used for queuing, modedesignation, and transfers of control. It is an atomic data structurewhich indicates the queue position of the current holder of the lock,the next available position in the queue of subsequent lock requests,the lock mode employed by the current successful lock requestor, andvalidation information which may be used to identify certain protocolfailures requiring lock overrides. Resources may also be provided in themain lock data structure to validate the identity of the successful lockrequestor in connection with certain transactions. In some aspects ofthe invention, the auxiliary lock data structure, 40, is used forvalidation and may be used to identify additional protocol failuresrequiring lock overrides, for example, those associated with aparticular lock mode. The auxiliary lock data structure, 40, may be asingle entry, the entry being a single atomic structure, or it may be anarray which includes an entry for each processor, each entry being asingle atomic structure. Each entry includes the resources needed toidentify the successful lock requestor's place in a queue of requestorsand to identify the successful lock requestor. Each processor typicallyinvokes its lock services procedure, for example procedure 6 b forprocessor 1 b, before starting a transaction on the shared resource 4,and may obtain a lock on the shared resource 4 if it is available. Onlyafter a successful requestor from among the processors obtains the lockwill that processor perform its transaction on shared resource 4. If theshared resource 4 is already locked at the time the request is received,or if there are multiple simultaneous requests for access, the lockservices procedure will queue the unsuccessful requests on a lockrequest queue 50. In relevant part, each of the lock services procedures6 a-6 n incorporates, in accordance with the present invention, a lockcontention procedure, at least two lock mode procedures, procedures forlocking, mode designation and unlocking operations by a successful lockrequester in normal operation, algorithms for arbitrating among multiplerequests for locks on the shared resource 4 from multiple unsuccessfulrequestors 1 a-1 n, and a polling procedure for allowing a previouslyunsuccessful requestor to determine its current status, and, in someaspects, lock override procedures and supplemental lock validationprocedures associated with various lock modes, all of which will befurther described below.

[0029] The shared resource 4 of computer system 10 may be almost anyresource that might be used by multiple processes, such as a massstorage device, a memory, a data structure within a memory, an ATM or acommunication device. The shared memory 2 of computer system 10 ismutually shared by or accessible to the processors 1 a-n. The sharedmemory 2 and shared resource 4 may be contained in a single logicalobject, in separate logical objects contained in a single physicalobject, such as two portions of a global memory, or they may be separatephysical and logical objects, such as a memory and a disk drive. In oneaspect, described in more detail below, the invention is implemented inan intelligent data storage system which includes several individualcomponents coupled via internal communications channels, and the sharedresource 4 is one or more of a set of shared data resources, such asdata records, data management records and blocks of data, in the datastorage system.

[0030] Referring now to FIG. 2 the computer system 10 of FIG. 1 is shownin more detail. Computer system 10 includes an intelligent data storagesystem 14, and may also include a plurality of host processors 12 a-12 nconnected to the intelligent data storage system 14 by hostcommunication channels 13 a-13(2 n). The storage system 14 includes aplurality of host controllers 21 a-21 n which are, according to apreferred embodiment of the present invention, coupled alternately tobuses 22 and 23. Each host controller 21 a-21 n is responsible formanaging the communication between its associated attached hostcomputers and storage system 14. Storage system 14 also includes aglobal memory 11 coupled to both buses 22 and 23. Preferably, the globalmemory is a high speed random access semiconductor memory. Global memory11 includes a large cache memory 15 which is used during the transfer ofdata between the host computers and the storage devices of arrays 26a-26 n. The global memory 11 also includes, as further described below,a cache manager memory 16 and a cache index directory 18 which providesan indication of the data which in stored in the cache memory 15 andprovides the addresses of the data which is stored in the cache memory.Also coupled alternately to buses 22 and 23 are a plurality of devicecontrollers 25 a-25 n. Coupled to each device controller is an array ofmass storage devices 26 a-26 n which as shown here may be magnetic diskdevices. As with the host controllers described above, each devicecontroller is responsible for managing the communications between itsassociated array of drives and the host controllers 21 a-21 n or globalmemory 11 of storage system 14.

[0031] A set of shared data resources in which data may be stored areimplemented in data storage system 14 and accessible by a plurality ofthe processors in system 10. Some or all of the data records, blocks ofdata and data management records in the global memory 11 and devicearrays 26 a-26 n may be shared data resources. By way of example and inorder illustrate certain aspects of the invention, the invention will beexplained by treating a single data structure implemented in a portionof global memory 11 as the only shared resource 4. The exemplary datastructure is a replacement queue 20, formed from a region of sharedmemory, such as cache manager memory 16. Replacement queue 20 isanalogous to the “least recently used” (LRU) queue used in prior artcache managers for readily identifying the least-recently-used dataelement in the cache. Because the cache memory has a capacity that issmaller than the main memory, it is sometimes necessary for dataelements in the cache memory to be removed from or replaced in the cachememory in order to provide space for new data elements being staged intothe cache memory. Typically, the cache manager will remove or replacethe “least-recently-used” data element in replacement queue 20. Varioustechniques have been described for dynamically monitoring and adjustingcache parameters, as described, for example, in the '473 patent and the'939 patent, supra. The performance of system 14 is highly dependent onthe cache management strategy selected. The strategy is implemented byprocedures 27 a-27 n. Since some of these strategies allow the cacheslot at the head of replacement queue 20 to contain something other thanthe “least-recently-used” data element, replacement queue 20 is referredto more generally as the replacement queue.

[0032] It will be understood, however, that the typical intelligent datastorage system 14 includes many such shared data resources. Theinvention is equally applicable to any shared resource 4 in a system 10which may be accessed by a plurality of the processors through a queuedlock. By way of example and not by way of limitation, other sharedresources in intelligent data storage system 14 may include cache indexdirectory 18, other data structures in cache manager memory 16, some orall of the data records in cache memory 10, and some or all of theblocks of data on disk arrays 26 a-26 n. Intelligent data storagesystems for certain applications, such as those supporting airlinereservation systems, may require extensive locking of shared dataresources, while other applications may require locking of fewer dataresources.

[0033] In the exemplary embodiment, the main lock data structure 30 andthe auxiliary lock data structure 40, further described in connectionwith FIG. 3 and FIG. 4, are also implemented in cache manager memory.Various procedures may be executed by each of the host controllers 21a-21 n and device controllers 25 a-25 n to access and manage thereplacement queue 20 as well as other shared data resources in cachememory 15, cache index directory 18 and cache manager memory 16, asfurther described, for example, in the '539 patent, the '307 patent, the'144 patent, and the '473 patent, all of which are herein incorporatedin their entirety by reference. Procedures 6 a-6(2 n) are the lockservices procedures of this invention. Procedures 27 a-27(2 n) are thereplacement queue management procedures for host controllers 21 a-21 nand device controllers 25 a-25 n respectively. Thus, in the illustrativeembodiment, the shared resource 4 is replacement queue 20 implemented inthe cache manager memory 16 of global memory 11, the processors 1 a-nare the host controllers 21 a-21 n and device controllers 25 a-25 n,processes A-N are the replacement queue management procedures 27 a-27(2n) which manage the replacement queue 20, and the shared memory 2 isalso the cache manager memory 16. The storage busses 22 and 23 provideaccess to the shared resource 4, so these are the first communicationchannels 3 a-3 n. The storage busses 22 and 23 also provide access tothe shared memory 2 so these are the second communication channels 7 a-7n. Local memory 5 a-5 n will typically be implemented on both hostcontrollers 21 a-21 n and device controllers 25 a-25 n.

[0034] It should be noted that this example illustrates two preferredaspects of the invention, namely, that the system embodying theinvention is the intelligent data storage system 14 and that theprocessors access the lock data structures 30 and 40 over the samechannels used to access the shared resource 4, i.e. the first and secondcommunication channels are identical. However, within the scope of theinvention, the processors may be any or all of the host controllers 21a-21 n, device controllers 25 a-25 n, or host computers 12 a-12 n, thechannels 3 a-3 n may be any or all of channels 13 a-13 n or busses 22 or23, and the processes A-N and associated lock services procedures 6 a-6n may be other processes or procedures managing other shared dataresources. Moreover, the lock data structures 30 and 40 need not residein the same logical device Or be accessed over the same channels as eachother or as the shared resource 4. The invention is also applicable toembodiments where the first and second communication channels areseparate.

[0035] Before proceeding further, it may be helpful to describe the datastructures used in one embodiment of the invention. FIG. 3 is aschematic diagram of a preferred form of the main lock data structure30, and FIG. 4 is a schematic diagram of a preferred form of theauxiliary lock data structure, 40.

[0036] MAIN, the main lock data structure, 30, is short enough for anatomic operation and typically has the following form:

[0037] HOLDER_ID, LOCK_MODE, LOCK_PW, CURRENT_HOLDER, NEXT_FREE.

[0038] The HOLDER_ID parameter, 31, may be used as an identifier of therequester which currently holds the lock. Each possible requestor in thesystem is assigned a unique HOLDER_ID. In some aspects of the invention,it is only updated in connection with certain lock modes, so it may notalways identify the current holder of the lock. It is an optionalparameter, since it is used primarily to validate the identity of asuccessful lock requestor

[0039] The LOCK_MODE parameter, 33, specifies the type of lock which iscurrently being used by the current lock holder. In addition to thebasic lock mode procedure associated with a particular LOCK_MODEparameter, one or more supplemental validation procedures, lock overrideprocedures, or both may be selectively associated with each LOCK_MODEparameter. For example, some processor operations take much longer thanothers, and in systems which implement a preset timeout to override thelock in the event of a protocol failure, it may be desirable toestablish a lock mode for these longer operations in which the normaltimeout will not occur. Thus, a first lock mode may be associated with anormal timeout lock override procedure and a second lock mode with adifferent timeout procedure, or none at all. Additional lock modes maybe associated, for example, with shared access to certain data. In oneaspect of the invention, one of lock modes (and any lock mode,supplemental validation or override procedures associated with this lockmode) will be the default lock mode. In order to illustrate theinvention, and not to limit it, a dual-mode locking system will bedescribed, and the only differences between the two lock modes will bethe supplemental validation and lock override procedures associated withthem. In the illustrative embodiment, the first lock mode is associatedwith a competitive, normal timeout lock override procedure and has nosupplemental validation procedure, while the second lock mode does havean associated supplemental validation procedure and is also associatedwith two lock override procedures, one a competitive, long timeoutprocedure and the other a cooperative, event-based lock overrideprocedure. However, if one byte is allocated to the LOCK_MODE parameter,up to two hundred fifty-six lock modes, with their associated lock mode,supplemental validation and lock override procedures, may be supportedwithin the atomic data structure for MAIN. In the illustrativeembodiment the LOCK_MODE value for a normal timeout mode is the defaultsetting “0” for the LOCK_MODE parameter, while “T” is the LOCK_MODEvalue for long timeout.

[0040] The LOCK_PW parameter, 35, indicates whether a valid lock isheld. It has a valid value for the “no lock holder” state, and one ormore valid values indicating that the lock is held. All other values areinvalid. In one aspect of the invention, each shared resource, 4, isassigned its own value of LOCK_PW, 35. This parameter may be used toidentify certain protocol failures requiring lock overrides.

[0041] The CURRENT_HOLDER parameter, 37, indicates which place in thelock request queue presently holds the lock. It indicates a place inline, not an identification, but, as will be explained below, it enablesthe requestor which holds that place in line to determine when it maytake the lock.

[0042] The NEXT_FREE parameter, 39, indicates the next available placein the lock queue. Both CURRENT_HOLDER and NEXT_FREE are numericparameters whose values wrap so that the allowable size of the parameteris never exceeded.

[0043] AUX, the auxiliary lock data structure, may be a single entry,the entry being a single atomic structure, or it may be an array whichincludes an entry for each processor, each entry being a single atomicstructure. In the embodiment shown, AUX, 40, is a single entry, shortenough for an atomic operation, and typically has the following form:

[0044] MY_ID (optional), LOCK_MODE_AUX, MY_NUMBER_AUX,

[0045] TIME_STAMP_AUX (optional).

[0046] Since the auxiliary lock data structure, 40, is used primarily toassist in determining when a protocol failure requiring certain lockoverride procedures has occurred, it is typically not updated every timea new requestor takes the lock. This feature of the invention will befurther described in connection with FIG. 7.

[0047] The MY_ID parameter, 41, is an identifier uniquely associatedwith each processor. As will be further discussed below, the entry istypically refreshed only when that processor is the requestor whichcurrently holds the lock, and only in connection with certain lockmodes. In the array form of AUX, only one value of MY_ID(i) is valid forany given entry, since each entry is associated with and can be writtenby only one processor, but in the illustrated form, N different valuesof MY-ID are valid, one being associated with each of the N possiblerequesters, This parameter is optional, but may be used for validationin certain protocol failure situations, as further explained below.

[0048] The LOCK_MODE_AUX parameter, 43, specifies the type of lock whichis currently being used by the current lock holder. It has the samepossible values and serves the same purpose as the LOCK_MODE parameter,53.

[0049] The MY_NUMBER_AUX parameter, 45, indicates what place in thequeue the processor holds. The entry is typically refreshed only inconnection with certain lock modes when a requestor which holds the lockin that mode. In the array form of AUX, each processor may refresh onlythe value in its own entry in the array.

[0050] The TIME_STAMP_AUX parameter, 47, indicates the time at which theprocessor making the entry obtained the lock. It is typically used tostart a timeout clock. This parameter is optional, but may be used forcertain types of lock overrides, as will be further explained below.

[0051] In addition to MAIN, 30, and AUX, 40, which must be stored inshared memory, 2, so that all possible requestors may access them, twoadditional numerical variables, MY_NUMBER, 51 a-n, and TIME_STAMP_L, 53a-n, are associated with each potential requestor. While these may bestored in any system resource to which the requestor has access,typically, both MY_NUMBER, 51 i, and TIME_STAMP_L, 53 i, are stored inthe local memory associated with each potential requestor in order toreduce bus traffic. Each requester also requires sufficient local memoryto store the two most recent values of MAIN and the value of an AUXentry.

[0052] Turning now FIG. 5, the steps used to enter the lock requestqueue, and to poll for and obtain the lock during normal operation ofthe system described herein are illustrated in a flowchart. Prior toentering the process described in FIG. 5, processor 1 a has, in thecourse of executing process A, identified a need to obtain a lock on ashared resource 4, illustratively, the replacement queue, 20. In asingle atomic read-modify-write operation, represented on the flowchartby steps 100, 102, 104, and 106, processor 1 a initiates its attempt toobtain the lock. In step 100, processor 1 a reads MAN, and in step 102,determines whether the lock is validly held. If the lock is currentlyheld by another requestor, the LOCK_PW, 35, will have a valid valueindicating that the lock is held.

[0053] If this condition is not true, processor 1 a will reserve thelock in default mode and establish a lock request queue at step 106 bysetting HOLDER_ID, 31 to its own value, LOCK_MODE, 33 to “0”, LOCK_PW,35, to a valid value, CURRENT_HOLDER, 37, to the value presently enteredin NEXT_FREE, 39, and by incrementing NEXT_FREE, 39. Next, at step 131,processor 1 a makes a good exit to process A. Processor 1 a may call thesupplemental validation process described in connection with FIG. 7,either immediately upon completing step 106, if it requires the lock ina mode other than the default mode, or at some later point in itsexecution of process A, if, for example, an error or branch conditioncreates the need for an alternate activity, like recovering thestructure of the shared resource, which would require the alternate lockmode.

[0054] Assuming that the lock is validly held by another processor,processor 1 a queues for the lock in a single atomic read-modify-writeoperation represented in FIG. 5 by steps 100, 102 and 104. If uponreading MAIN in step 100, processor 1 a determines that the lock isvalidly held by another requester by the method previously described inconnection with step 102, then, at step 104, processor 1 a will reservethe next available number in the queue by incrementing the value ofNEXT_FREE in MAIN. At step 108, processor 1 a enters the queue bysetting the value of MY_NUMBER, 51 a, to the value of NEXT_FREE it readin step 102.

[0055] The processor then updates the timeout parameters at step 110,assuming the lock mode it detected in step 100 by reading MAIN has atimeout-based lock override procedure associated with it in lockservices procedure 6. If there is no timeout-based lock overrideprocedure associated with the lock mode, then processor 1 a may jumpdirectly to the lock polling sequence beginning at step 118. In theexemplary embodiment shown in FIG. 5, there is a timeout-based lockoverride procedure associated with each of the two possible lock modes,so at step 110, processor 1 a updates in its local memory the overrideparameters associated with the lock mode it has found to be in effect.Each lock mode which has an associated timeout procedure may use adifferent source for its reference value and a different predeterminedinterval associated with it in lock services procedure 6. Thus, forexample, the normal timeout mode may use obtain its reference value fromits own clock and have a timeout interval of a few seconds or less,while the long timeout mode may obtain its reference value from AUX, 40,and have a timeout interval of many minutes. So, in one aspect of theinvention, processor 1 a performs the update by saving the time at whichstep 108 occurs (as measured by its own internal clock) in TIME_STAMP_L,53, for use as a reference value in monitoring whether a timeout hasoccurred. In this approach, the timeout is established and monitoredwithout involving scarce system resources such as the busses in anyadditional I/O cycles, so it is suitable for use as the lock overrideprocedure corresponding to the default lock mode. In another aspect ofthe invention, processor 1 a may perform this update by taking atimestamp value from TIME_STAMP_AUX, 47 for use as a reference value inmonitoring whether a timeout has occurred. If AUX is an array, Processor1 a determines what entry in AUX to use for this purpose from the valueof HOLDER_ID, 31, which processor 1 a read in MAIN, 30, at step 100. Forvalidation, processor 1 a may confirm that its LOCK_MODE_AUX is set tothe second lock mode, and, if MY_D is implemented, may confirm that AUXalso has a value of MY_ID corresponding to the value of HOLDER_ID. If,when processor 1 a executes these validation steps, AUX is found not tobe valid, processor 1 a may default to a short, fixed, timeout value. Ifa valid AUX entry is found, processor 1 a will save the time fromTIME_STAMP_AUX to the processor's local memory, for example inTIME_STAMP_L for use in monitoring whether a timeout has occurred. Inthis aspect of the invention, several additional I/O cycles involvingscarce system resources are required to validate the lock mode andestablish the reference value for the timeout, so this approach is mostsuitable when either the timeout procedure itself or the lock modeprocedure it is associated with (or both) are expected to consume manymore I/O cycles or system resources than the default lock mode. In thissituation, the small number of I/O cycles used may be justified bydecreased likelihood that one or both of these procedures will beinitiated in error.

[0056] Assuming that a timeout-based lock override has been determinedto be associated with operative lock mode, in step 116, processor 1 awill continue with the procedure by testing to see if a timeout hasoccurred by determining whether the predetermined interval has elapsedsince the reference value for the timeout was updated. If a tout isdetected, at step 130, processor 1 a enters the lock forcing processfurther described in connection with FIG. 6. If a timeout has notoccurred, processor 1 a begins polling MAN. In one embodiment of theinvention, at step 118, processor 1 a estimates, before every repetitionof polling step 120, the number of prior entries in the lock requestqueue and adaptively delays its polling period as a function of saidnumber of prior entries in said lock request queue. The polling periodmay be estimated as the product of the number of significant processoroperations expected to be performed before processor 1 a obtains thelock as a function of the number of prior entries in said lock requestqueue and the average duration of a significant processor operationinvolving the shared resource. This delay procedure is further describedin U.S. Ser. No. 09/312,146 filed 14 May 1999 by Ofer et al and entitled“Adaptive Delay of Polling Frequencies in a Distributed System with aQueued Lock”, which is herein incorporated by reference in its entirety.

[0057] After polling MAIN in step 120, processor 1 a performs a sequenceof sanity checks on the updated value of MAIN, 30, which it has obtainedfrom the polling step, 120, and stored in its local memory. The sanitycheck sequence may also be entered from the lock forcing process of step130 after a failed attempt to force the lock. If processor 1 adetermines at step 122 that the LOCK_PW, 35, is invalid, processor 1 awill jump to step 100 and attempt to obtain the lock. If the LOCK_PW,35, is valid and processor 1 a finds at step 124 that it has obtainedthe lock, i.e. that the value of CURRENT_HOLDER, 37, read at step 120equals MY_NUMBER, 51 a, processor 1 a will enter the goodexit/supplemental validation process at step 131. If upon reading MAN,30, in step 120, processor 1 a determines at step 122 that the LOCK_PW,35, is valid and at step 124 that the lock is still held by anotherrequester by the method previously described in connection with step102, then, at step 126, processor 1 a compares MY_NUMBER withCURRENT_HOLDER and NEXT_FREE to determine whether processor 1 a is stillin the queue. If, when adjusted for the queue wrap, MY_NUMBER is notbetween CURRENT_HOLDER and NEXT_FREE, this indicates that the lock hasbeen reset due to a lock override, as will be described further inconnection with FIG. 6, and processor 1 a is not a member of the currentqueue of lock requestors. Processor 1 a then goes to step 100 andrepeats steps 100, 102, 104, and 108 in order to join the new lockqueue. If step 126 confirms that processor 1 a is still part of thecurrent lock request queue, then, as will be further discussed inconnection with the lock override procedures described below, at step128 processor 1 a will determine if CURRENT_HOLDER, 37, LOCK_MODE, 33,or LOCK_PW, 35 has changed. At step 129, processor 1 a may update itstimeout parameters if any of these has changed since its last reading ofMAIN.

[0058] In one aspect of the invention, each processor implements amonitoring procedure, M, for detecting a predetermined indication ofprotocol failure by an one of the plurality of processors andidentifying the failing processor. This procedure, M, is external tolock contention procedure, but may be used to trigger certain lockoverride procedures, for example, the cooperative lock override processdescribed in connection with FIG. 8. In one aspect of the invention,shown in FIG. 5, processor 1 a will determine at step 112 whether thelockholder is operating in a lock mode associated with an override whichuses this trigger, such as the cooperative lock override, and if thelockholder is, process May be periodically polled by processor 1 a uponreceiving an indication of protocol failure during this poll, processor1 a will initiate a lock override process at step 114, as furtherdescribed in connection with FIG. 8. At the conclusion of the processshown at step 114, there will typically be a new lockholder, andprocessor 1 a will go to step 116 to continue checking for timeouts.Alternatively, procedure M may cause a jump to a lock override processat step 114, as further described in connection with FIG. 8. Theprocedure M is shown for convenience operating at step 113 although itwill be understood that it operates periodically so long as theprocessors are running. In a preferred embodiment, the polls do notoccur when the lockholder is operating in the default lock mode, butonly in connection with a more resource-intensive lock mode such as thelong timeout mode. However, in an alternative embodiment, polls of orjumps to and from process M may occur at any time in the course of lockcontention procedure.

[0059] If processor 1 a determines that the present lock mode is notassociated with process M at step 112, or if no protocol failure isindicated by process M in step 113, processor 1 a will continue checkingfor timeouts at step 116. So long as processor 1 a does not obtain thelock and no lock override is initiated as described below in connectionwith FIG. 6 or FIG. 8, processor 1 a repeats the applicable steps in thelock polling sequence, 116 through 120, and the subsequent sanity checksequence 122 through 129 (with steps 112, 113, and 114 if the lock modeso requires), until it determines either that LOCK_PW, 35, has aninvalid value or that MY_NUMBER, 51 a, equals CURRENT_HOLDER, 37, eitherof which cause it to take the lock and make a good exit to process A, asdescribed in steps 106 and 131, or it determines that a timeout or otherevent requiring a lock override has occurred. Various procedures forhandling lock overrides are discussed in connection with FIGS. 6, 7 and8.

[0060]FIG. 6 is a flowchart illustrating steps used to perform a lockoverride procedure associated with a selected one of the lock modes inthe system described herein. This lock override procedure is a timeoutprocedure. Different timeout procedures with different reference valuesand timeout intervals may be associated with different lock modes. Inthe exemplary embodiment, for example, a normal, i.e. short, timeoutinterval using a first reference value is associated with the default“0” lock mode and a long timeout interval using a second reference valueis associated with the other “T” lock mode Referring now to step 116 inFIG. 5, processor 1 a tests to see if a timeout has occurred bydetermining whether a predetermined interval has elapsed since thereference value for the timeout. If a timeout has occurred, processor 1a enters the lock forcing process of step 130. Going now to FIG. 6,where the process of 130 is illustrated in more detail, if processor 1 adetermines that a timeout has occurred, then, entering the lock forcingprocess at Y upon setting a hardware lock, then, in a single atomicread-modify-write operation, represented on the flowchart by steps 132,1347 and 136, processor 1 a initiates its attempt to obtain the lock Atstep 132, processor 1 a will read MAIN, 30, and at step 134 willdetermine whether MAIN, 30, has changed since processor 1 a last readMAIN and stored its value in local memory. If it has not, at step 136,processor 1 a will force the lock, and reset the entire lock requestqueue, by setting CURRENT_HOLDER, 37, equal to value of NEXT_FREE, 39,it read in step 132, incrementing NEXT_FREE, setting the LOCK_MODE, 33,to the default mode indicator (regardless of which lock mode processor 1a actually requires), setting HOLDER_ID, 31, to its own identifier andsetting the LOCK_PW, 35, to a valid password. Steps 132, 134, and 136must be performed as an atomic operation. At step 138, processor 1 awill complete the lock override procedure by setting MY_NUMBER, 51 a,equal to the value of NEXT_FREE, 39, it read in step 132. Processor 1 awill then make a good exit to process A. As discussed in connection withstep 131 in FIG. 5, should processor 1 a require the lock in some modeother than the default mode, it will, as a part of this process, proceedas described in connection with FIG. 7. Otherwise, it will simply takethe lock and exit the lock contention procedure.

[0061] If more than one processor is in the lock request queue when thefirst mode timeout occurs, it is possible that more than one processorwill detect the event and attempt to force the lock. It would beundesirable for more than one processor to do so successfully. So, ifprocessor 1 a detects in step 134 that MAIN, 30, has changed since thelast time processor 1 a polled MAIN, it will release its hardware lockat Z and exit the forcing procedure. It will then continue with thesanity check sequence described in connection with FIG. 5, beginningwith step 122, if implemented, using the new value of MAIN which it readat step 132, and proceeding to steps 124 and beyond. Typically, in thisscenario, processor 1 a will detect in step 126 that the lock requestqueue has been reset, and will then repeat steps 100, 102, 104, and 108in order to join the new lock queue. If processor 1 a has not detectedthe timeout before the lock is forced, and so never enters the lockforcing process, then when processor 1 a reaches step 126 in its regularpolling sequence, it will detect that MY_NUMBER, 51 a, is no longer inthe queue and will also repeat steps 100, 102, 104, and 108 in order tojoin the new lock queue.

[0062]FIG. 7 is a flowchart illustrating steps used to select a lockmode (in this case, the second lock mode) other than the default lockmode, to perform a supplemental validation associated with the selectedlock mode, and to initialize a second lock override procedure associatedwith the selected lock mode. FIG. 8 will describe how the second lockoverride procedure is performed. The second lock override procedure is acooperative lock override procedure, and, for purposes of illustration,will be associated with the second, or long timeout lock mode. Becauseit involves a number of steps using scarce system resources, thecooperative lock override procedure is most suitably associated with alock mode expected to consume many more I/O cycles or system resourcesthan the default lock mode. To minimize the likelihood of tying up thesesystem resources in error, a supplemental validation procedure isselectively associated with this lock mode. For purposes of thisdiscussion and the one that follows in connection with FIG. 9, it willbe assumed that processor 1 a has queued for the lock and determined instep 124 of FIG. 5 that its MY_NUMBER, 51 a, corresponds toCURRENT_HOLDER, 37. Processor 1 a has therefore made a good exit toprocess A at step 131. It will also be assumed that processor 1 f isnext in the lock request queue.

[0063] Turning now to FIG. 7, where the supplemental validation processis described in more detail, processor 1 a calls the supplementalvalidation process from process A, as discussed in connection with step131 of FIG. 5, because it needs an alternate mode for which supplementalvalidation is associated, in this case the long timeout mode. In step140, processor 1 a updates AUX, 40 by setting LOCK_MODE_AUX, 43, to theidentifier of the lock mode it requires, in this case the identifier,“T”, for long timeout mode, and MY_NUMBER_AUX, 45, to MY_NUMBER, 51 a,the number of its place in the queue. If AUX is an array, processor 1 awill update only the values in its own entry AUX(a). It should be notedthat TIME_STAMP_AUX, 47, and MY_ID, 41, are not required parameters inconnection with the second lock override procedure illustrated in FIG.7, although either or both may optionally be used for validation inconnection with this procedure. If a timeout is associated with theselected lock mode, or if TIME_STAMP_AUX, 47, is to be used forvalidation, processor 1 a will also update TIME_STAMP_AUX, 47, to thetime at which step 140 occurs, and if MY_ID, 41, is implemented in AUX,will update MY_ID to the value of its unique identifier. It is notnecessary to implement a timeout in addition to the cooperative lockoverride procedure described below in connection with any selected mode,but depending on the events used to trigger the cooperative lockoverride procedure, it may be desirable to do so. In the illustrativeembodiment, as will be discussed in connection with FIG. 8, both anevent-based cooperative lock override procedure and a timeout-based lockoverride procedure are associated with the long timeout mode. Typically,if both are implemented, processor 1 a reads an internal clock,preferably the system clock, to determine the time at which step 140occurred and puts this value in TIME_STAMP_AUX, 47. In an atomicread-modify-write operation shown as steps 142, 144, and 146, processor1 a then reads MAIN, 30, at step 142 and determines at step 144 whetherit validly holds the lock by determining whether MY_NUMBER, 51 a, isequal to CURRENT_HOLDER and the LOCK_PW, 35, has a valid value. Sinceprocessor 1 a has just taken the lock, in the absence of a memorycorruption involving MAIN, 30, or other protocol error, this operationis expected confirm its custody of the lock. Upon receiving confirmationthat it still holds the lock and still as part of the atomic operationbegun in step 142, processor 1 a updates MAIN, 30, in step 146 bysetting LOCK_MODE, 33, to the mode indicator “T”, and updating thevalidation parameters implemented in MAIN. Processor 1 a then exits thesupplemental validation process at 151 and proceeds with process A. Ifany confirmation step in the sequence fails to confirm that processor 1a holds the lock, then processor 1 a gives a “bad status” error messageto process A at step 148 and exits the lock contention process toprocess A at step 150, relinquishing any hardware locks as it does so.Although each confirmation requires an extra bus cycle, any failure toconfirm is strong evidence of a protocol violation involving processorholding the lock or the lock itself. Once a resource is locked into along timeout mode (or another high I/O demand mode) in error, detectingand correcting the problem typically requires a great many bus cycles tocorrect. The validation steps significantly decrease the likelihood ofsuch errors.

[0064] The processes indicated at steps 113 and 114 of FIG. 5 will nowbe described in more detail. Turning now to FIG. 8, the second lockoverride procedure is initiated at M when any processor detects certaintypes of protocol failures while the processor which hold the lock isoperating in a lock mode associated with the second lock overrideprocedure, by way of example, the long timeout mode. The detectingprocessor need not be a current member of the lock request queue, andmay, in some instances, even be the one which holds the lock. In oneaspect, the second lock override procedure may be initiated when aprocessor receives a predetermined indication from a process M externalto the lock services procedure that another processor is malfunctioning.For example, in the SYMMETRIX® storage systems manufactured by EMCCorporation, the processors monitor certain of their own functions. If aprocessor detects certain types of malfunctions, it will put a messagein a first predetermined area in global memory indicating that it ismalfunctioning. All processors periodically poll this area forindications of malfunctions in the other processors. In addition, eachprocessor periodically sends a signal, called a heartbeat, over thecommon bus to a predetermined area in global memory to indicate that itis in good working order, and all processors monitor the heartbeats ofall other processors by polling for these heartbeats. If the pollingprocessor fails to detect the heartbeat of another processor for apredetermined interval, the polling processor determines that the silentprocessor has malfunctioned. A processor may even trigger this overrideon itself as lockholder if it receives an indication, for example, fromprocess A, that it may have failed to clear the lock in a previousoperation, Other events can also be used to trigger the cooperativeoverride procedure. At step 113 in FIG. 5, processor 1 c detects amalfunction in processor 1 a via process M and enters the cooperativeoverride process shown at step 114 in FIG. 5. At step 152, processor 1 creads AUX, 40, or, if AUX is an array, AUX(a) corresponding to processor1 a At step 154, processor 1 c determines whether processor 1 a had setits LOCK_MODE_AUX entry, 43, to indicate a mode associated with thecooperative lock override procedure, in our example, the long timeoutmode. The value of MY_NUMBER_AUX, 45, in AUX, 40, indicates what placein the queue a processor held the last time it updated AUX. However,since each processor updates its entry in AUX only when it requires along timeout mode and corruption of the data in the interim periods ispossible, it is desirable to validate AUX, 40, using either the time inTIME_STAMP_AUX, 47, or the processor identifier in MY_ID, 41, or both.If the entry is corrupt, it is unlikely that MY_ID will contain theproper identifier and if the entry is outdated, the time inTIME_STAMP_AUX(a) will so indicate. Since each AUX entry is atomic, allof the reads necessary for validation require only one bus I/O cycle. IfAUX is not validated or does include the indicator for the long timeoutmode, the second lock override procedure will not be implemented, andprocessor 1 c will exit the sequence at 168. If AUX indicates thatprocessor 1 a held the lock in long timeout mode, and the requisitevalidation criteria are satisfied, then at step 156, processor 1 c readsMAIN, 30, and at step 158 attempts to validate that processor 1 a heldthe lock in long timeout mode. If MAIN, 30 is not validated or doesinclude the requisite indicators for the long timeout mode, the secondlock override procedure will not be implemented, and, as before,processor 1 c will exit the sequence at 68. If processor 1 c is notqueued for the lock, at 168 it will exit the lock contention procedure,but if processor 1 c is a member of the lock request queue, from 168 itwill continue the lock polling sequence at step 116 in FIG. 5.

[0065] If at step 158, processor 1 c does validate that processor 1 aheld the lock in long timeout mode by finding that CURRENT_HOLDER, 37,has the same value as the value of MY_NUMBER_AUX, 457 which processor 1a read in step 152, that LOCK_MODE, 33, is set to indicate the longtimeout mode, and that LOCK_PW, 35, and HOLDER_ID, 31, if implemented,validate that processor 1 a holds the lock, then at step 160, processor1 c determines whether the value of NEXT_FREE, 39, read at step 156 isequal to CURRENT_HOLDER, 37, plus 1. If it is, there is no otherrequestor in the queue, so at step 162, processor 1 c updates MAIN toindicate the lock is not held by setting CURRENT_HOLDER, 37, equal tothe value of NEXT_FREE, 39, setting the LOCK_MODE, 33, to its defaultvalue and setting the LOCK_PW, 35, to indicate “no lock holder”. IfNEXT_FREE, 39, is not equal to CURRENT_HOLDER, 37, plus 1, there areother requestors in the lock queue, so at step 164, processor 1 cupdates MAIN, 30, by incrementing CURRENT_HOLDER, 37, setting LOCK_MODE,33 to its default value and setting the LOCK_PW, 35, to any valid value.Following step 162 or 164, at step 166, processor 1 c invalidates AUX,40, by writing over at least MY_ID, 41, and preferably the entire entry,and then exits the cooperative lock override procedure at step 168, asdescribed above. Meanwhile, the processors in the lock request queuewill continue with the lock polling sequence described in connectionwith FIG. 5. Processor 1 f, the lock requestor which has been moved tothe head of the queue by processor 1 c will detect on its next poll thatthe LOCK_PW, 35, is valid and that MY_NUMBER, 51 c, is now equal toCURRENT_HOLDER, 37, and will accept the lock.

[0066] Turning now to FIG. 9, the procedure for unlocking the lock inthe absence of a protocol error is shown. As indicated above inconnection with FIG. 8, it will be assumed that processor 1 a holds thelock in long timeout mode and processor 1 f is the next requestor in thequeue. Except where indicated, the steps are the same regardless ofwhether processor 1 a held the lock in default mode or in another mode.It will also be assumed that processor 1 a has successfully completedthe portion of process A which required a lock on the shared resource 4and still retains the lock, i.e. that no other processor has completed alock override procedure. At step 170, processor 1 a reads MAIN, 30, andat step 172 determines whether MAIN is valid and whether the value ofCURRENT_HOLDER, 37, read at step 170 is equal to the value of MY_NUMBER,51 a. If both conditions are satisfied, then at step 174, processor 1 adetermines whether the value of NEXT_FREE, 39, read at step 170 is equalto CURRENT_HOLDER, 37, plus 1. If it is, there is no other requestor inthe queue, so at step 176, processor 1 a updates MAIN to indicate thelock is not held by setting CURRENT_HOLDER, 37, equal to the value ofNEXT_FREE, 39, setting the LOCK_MODE, 33, to its default value andsetting the LOCK_PW, 35, to indicate “no lock holder”. If NEXT_FREE, 39,is not equal to CURRENT_HOLDER, 37, plus 1, there are other requestersin the lock queue, so at step 178, processor 1 a updates MAIN, 30, byincrementing CURRENT_HOLDER, 37, setting LOCK_MODE, 33 to its defaultvalue and setting the LOCK_PW, 35, to any valid value. These steps areperformed as an atomic read-modify-write operation. Following step 176or 178, or following step 172 if either of the conditions are notsatisfied, processor 1 a decides, at step 180 if it held the lock in alock mode associated with a lock override procedure which requires areference to AUX, 40, such as the cooperative lock override proceduredescribed in connection with FIG. 8, or a timeout-based procedure whichuses TIME_STAMP_AUX, 47, as its reference value. If it did not hold thelock in such a mode, it will exit the lock services procedure, 6 a, toresume process A. However, in the exemplary embodiment, processor 1 aheld the lock in long timeout mode, which is associated with both thecooperative lock override procedure and a timeout procedure which usesTIME_STAMP_AUX, 47, as its reference value, so at step 182, processor 1a invalidates AUX by writing over at least MY_ID, and preferably theentire entry, and then exits the lock services procedure to resumeprocess A. Meanwhile, processor 1 f, continuing with the lock contentionprocedure of FIG. 5, will shortly discover at step 124 thatCURRENT_HOLDER, 37, is now equal to MY_NUMBER, 51 f, and so, in normaloperation, the lock will pass to the next member of the queue.

[0067] Having described a preferred embodiment of the present invention,it will now become apparent to those of skill in the art that otherembodiments incorporating its concepts may be provided. It is felttherefore that this invention should not be limited to the disclosedembodiment but rather should be limited only by the spirit and scope ofthe following claims.

1. A method for providing cooperative queued locking and unlockingservices for managing a shared resource in a data processing systemincluding a plurality of processors as lock requesters, each processorsupporting atomic operations and being coupled to the shared resourcethrough one or more first common communication channels, including thesteps of: providing for each processor a monitoring procedure fordetecting a predetermined indication of protocol failure by an one ofthe plurality of processors and identifying the failing processor;providing for each processor a lock services procedure including aqueuing procedure for unsuccessful lock requestors, locking andunlocking procedures for locking and unlocking the shared resource by asuccessful lock requestor, and a lock override procedure responsive tothe detection of the predetermined indication of protocol failure;providing for the shared resource, an associated main lock datastructure stored in a shared memory accessible by the plurality ofprocessors, the main lock data structure including in a single atomicstructure, the resources needed to lock the shared resource by asuccessful lock requester, to establish a queue of unsuccessful lockrequestors, and to validate the existence of the lock; providing for theshared resource, an associated auxiliary lock data structure stored in ashared memory accessible by the plurality of processors, the auxiliarylock data structure including the resources needed to identify thesuccessful lock requestor's place in a queue of requestors and toidentify the successful lock requestor; detecting, by one of theprocessors, a predetermined indication of protocol failure andidentifying the failing processor by the detecting processor; initiatingthe lock override procedure by the detecting processor responsive to thepredetermined indication of protocol failure; and, in a single atomicoperation by the detecting processor, examining the contents of theauxiliary lock data structure to determine if the identified failingprocessor is the successful lock requestor, and either, if theidentified failing processor is the successful lock requestor, in asingle atomic operation by the detecting processor, examining thecontents of the main lock data structure and writing data to the mainlock data structure to reserve the lock to the next requestor in thequeue of unsuccessful lock requesters and to revalidate the lock, or, ifthe identified failing processor is not the successful lock requester,exiting the cooperative lock override procedure.
 2. A method accordingto claim 1 wherein the lock services procedure further includes at leasttwo lock mode procedures and a lock mode selection procedure forselecting one from the lock mode procedures by a successful lockrequester, wherein the locking and unlocking procedures include one ormore procedures for locking and unlocking the shared resource in theselected lock mode by a successful lock requestor, wherein thecooperative lock override procedure is selectively associated with alock mode, wherein the atomic main lock data structure further includesthe resources needed to identify one of the lock modes, and wherein theauxiliary lock data structure further includes the resources needed toidentify one from the lock modes.
 3. A method according to claim 1wherein the method further includes, prior to examining the contents ofthe main lock data structure by the detecting processor, the step of ina single atomic operation by one of the requesting processors, examiningthe contents of the main lock data structure to determine if anotherrequesting processor has previously locked the shared resource and ifthe lock contents are valid, determining that the contents are invalidor no other requesting processor has previously locked the sharedresource, and writing data to the main lock data structure to reserveand validate the lock.
 4. A method according to claim 2 wherein, thestep of examining the contents of the main lock data structure by thedetecting processor includes the step of, in the same atomic operation,verifying by examining the contents of the main lock data structure thatthe identified lock mode is the lock mode associated with the first lockoverride procedure and the step of writing data to the main lock datastructure to reserve the lock to the next requestor in the queue ofunsuccessful lock requestors and to revalidate the lock, includes thestep of, in the same atomic operation, the step of invalidating theidentified lock mode.
 5. An intelligent data storage system comprising:a shared resource; a plurality of processors as lock requestors, eachprocessor supporting atomic operations and being coupled to the sharedresource through one or more first common communication channels, andeach processor implementing a monitoring procedure for detecting apredetermined indication of protocol failure by an one of the pluralityof processors and identifying the failing processor; a shared memoryaccessible over one or more second common communications channels to allof the processors; a lock services procedure implemented in each of theprocessors, the lock services procedure including a queuing procedurefor unsuccessful lock requestors, and locking and unlocking proceduresfor locking and unlocking the shared resource by a successful lockrequestor, and a cooperative lock override procedure responsive to thedetection of the predetermined indication of protocol failure; an atomicmain lock data structure, responsive to the lock services procedures,implemented in the shared memory and associated with the sharedresource, which includes the resources needed to lock a shared resourceby a successful lock requester, to establish a place in a queue ofunsuccessful lock requestors, and validate the existence of the lock; anatomic auxiliary lock data structure, responsive to the lock servicesprocedures, implemented in the shared memory and associated with theshared resource, which includes the resources needed to identify thesuccessful lock requestor's place in a queue of requesters and toidentify the successful lock requestors; each processor other than thesuccessful lock requester being operable in accordance with itsmonitoring procedure to detect a predetermined indication of protocolfailure and identify the failing processor; and in accordance with itslock services procedure, first to initiate its cooperative lock overrideprocedure responsive to its detection of the predetermined indication ofprotocol failure, and then in a single atomic operation, to examine thecontents of the auxiliary lock data structure to determine if theidentified failing processor is the successful lock requester, andeither, if the identified failing processor is the successful lockrequestor, in a single atomic operation, to examine the contents of themain lock data structure and write data to the main lock data structureto reserve the lock to the next requester in the queue of unsuccessfullock requestors and to revalidate the lock, or, if the identifiedfailing processor is not the successful lock requestor, to exit thecooperative lock override procedure.
 6. An intelligent data storagesystem according to claim 5 wherein the lock services procedure furtherincludes at least two lock mode procedures, a lock mode selectionprocedure for selecting one from the lock mode procedures by asuccessful lock requester, wherein the locking and unlocking proceduresinclude one or more procedures for locking and unlocking the sharedresource in the selected lock mode by a successful lock requestor,wherein the cooperative lock override procedure is selectivelyassociated with a lock mode, and wherein the atomic main lock datastructure includes the resources needed to identify one of the lockmodes.
 7. A method according to claim 6 wherein each requestingprocessor is further operable in accordance with its lock servicesprocedure to verify in examining the contents of the main lock datastructure in an atomic operation that the identified lock mode is thelock mode associated with the cooperative lock override procedure and toinvalidate the identified lock mode.
 8. An intelligent data storagesystem according to claim 5 wherein each requesting processor is furtheroperable in accordance with its lock services procedure, in a singleatomic operation, to examine the contents of the main lock datastructure to determine if another requesting processor has previouslylocked the shared resource and if the lock contents are valid, andeither, if the lock contents are valid and some other requestingprocessor has previously locked the shared resource, to write data tothe main lock data structure to establish its place in a queue ofrequestors for subsequent locks on the shared resource, or if thecontents are invalid or no other requesting processor has previouslylocked the shared resource, to write data to the main lock datastructure to reserve and validate the lock.